feat: Add redirected actor logs #403

Pijukatel · 2025-05-13T13:38:40Z

Description

Add convenience methods and arguments for redirecting streamed actor log into another logger.
This changes the default behavior in non breaking way. By default the logs will be redirected now.
Update log endpoints calls to accept raw parameter.

Example usage:

Starting new actor run through call

3 different use-cases:

...
async with Actor:
    # Default logger
    await Actor.new_client().actor(actor_id="...").call(timeout_secs=5)

    # No logger
    await Actor.new_client().actor(actor_id="...").call(timeout_secs=5, logger = None)

    # Custom logger
    custom_logger = logging.getLogger("apify.kustom_logga")
    handler = logging.StreamHandler()
    custom_logger.addHandler(handler)
    custom_logger.propagate = False

    await Actor.new_client().actor(actor_id="SbjD4JEucMevUdQAH").call(timeout_secs=5, logger = custom_logger)

Attaching to already running actor, redirecting all logs from actor start

(This is the technique to be used when starting actor through actor.start as well)

...
async with Actor:
    ...
    # With context manager
    async with await Actor.new_client().run("some_id").get_streamed_log():
        await asyncio.sleep(5)
        # Logging will stop out of context

     # With direct call (lifecycle of logging has to be manually handled)
    streamed_log = await Actor.new_client().run("some_id").get_streamed_log()
    streamed_log.start()
    await asyncio.sleep(5)
    streamed_log.close()

Attaching to already running actor, redirecting only new logs

(This can be useful when communicating with some long-running standby actor)

...
async with Actor:
    # With context manager
    async with await Actor.new_client().run("some_id").get_streamed_log(from_start=False):
        await asyncio.sleep(5)
        # Logging will stop out of context

Example actor with all the logging options shown:

https://console.apify.com/actors/m7ZFYX42SCRFOL0JW

Issues

Closes: Redirect logs from other actors #402

TODO: Message inspection to split bulk messages TODO: Log level guessing outside of filter

Update test to be able to use caplog

Pijukatel · 2025-05-13T14:05:09Z

~~This is a minimal draft version with only async client updated so far to have early review on the concept (sync client will be done later).~~

~~Such change can be later propagated to Actor class in sdk.~~

~~Example usage shown on the screenshot, where we call actor called logger three times with different logging arguments:~~

1. Default - redirect logger (will be new default behavior), user do not need to do anything and some predefined reasonable logger is used. (Logger name is f"apify.{actor}-{run_id}")
2. No redirect logger (same as current behavior, but have to be selected by explicitly passing None to logger argument)
3. Custom logger defined by the user (I have created some example logger for the sake of the example, but it can be anything.)

~~Redirected messages have estimated logging level, so in most cases you can figure out what level it was in the original log and set it accordingly in this log.~~

~~Example actor: https://console.apify.com/actors/m7ZFYX42SCRFOL0JW/source~~

Not in the scope of first version: Respecting the coloring of the original logs -> This information is not available in the logs returned from the api call, so we could try to recreate the coloring during runtime by inspecting all the messages and injecting color markers where suitable, but that could be very computational heavy, so not sure if it is worth it. It would be nice if api could return the colored original, but that would require saving the logs twice, which is also probably not worth it.

Refactor tests and code to reuse code.

Pijukatel · 2025-05-14T15:24:01Z

Original log colors are now respected:

Thanks a lot @barjin for letting me know about the secret parameter of the log endpoint!

Update default logger to not duplicate handlers if already exists

Copilot

Pull Request Overview

This PR introduces a new log redirection feature for Actor runs, allowing logs to be streamed and redirected to either a default or a custom logger. Key changes include:

New synchronous and asynchronous methods (get_streamed_log) in the Run and Log clients for redirecting logs.
Implementation of StreamedLog classes (sync and async) to handle chunked log messages and ensure proper formatting.
Updates to the Actor client to support an optional logger parameter for redirecting logs, along with additional tests and dependency updates.

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/test_logging.py	Added tests to verify correct log formatting and redirection behavior.
src/apify_client/clients/resource_clients/run.py	Introduced get_streamed_log methods for both sync and async clients.
src/apify_client/clients/resource_clients/log.py	Updated log retrieval methods with a new raw option and added StreamedLog classes for log streaming.
src/apify_client/clients/resource_clients/actor.py	Extended Actor.call method to support log redirection via an optional logger parameter.
src/apify_client/_logging.py	Added create_redirect_logger and improved RedirectLogFormatter with color support.
pyproject.toml	Added necessary dependencies for color support via colorama.

src/apify_client/clients/resource_clients/actor.py

src/apify_client/clients/resource_clients/log.py

Have explicit start and stop instead of __call__

Pijukatel · 2025-05-15T09:25:57Z

No need to do deep review @MQ37. I think you might be using this functionality soon. Could you please take a look at the use-cases and see if it is convenient to use from your point of view and if there is missing something?
(Timeouts and status messages will be done in separate PR)

src/apify_client/clients/resource_clients/log.py

janbuchar

I haven't managed to read the whole thing yet, but here are some comments

src/apify_client/clients/resource_clients/log.py

pyproject.toml

janbuchar · 2025-05-15T09:48:58Z

src/apify_client/clients/resource_clients/actor.py

+            logger: Logger used to redirect logs from the Actor run. By default, it is set to "default" which means that
+                the default logger will be created and used. Setting `None` will disable any log propagation. Passing
+                custom logger will redirect logs to the provided logger.


By default, it is set to "default" which means that the default logger will be created and used

Um, and what does that actually mean for the caller? Let's try and come up with a better description.

I tried to rephrase it like this:

.... Using "default" literal means that a predefined default logger will be used. ...

janbuchar · 2025-05-15T10:00:51Z

src/apify_client/clients/resource_clients/run.py

            **self._sub_resource_init_options(resource_path='log'),
        )

+    async def get_streamed_log(


I'm confused with the intended usage here. Are you supposed to call this method and then pass the result into an actor.call(logger=...) call?

No, actor.call(logger=...) is one group of use-cases that deal with you starting the other actor and then get_streamed_log is called internally.

Using get_streamed_log method directly has different use-case. It is when you want to stream log from already started actor (for example long-running actor in standby)

async with await Actor.new_client().run("some_id").get_streamed_log(from_start=False): # Do stuff while logs from other actor are redirceted

MQ37

Just checked the example usage and from UX viewpoint this looks good, thank you 👍 @Pijukatel

…byte character

vdusek

Does the Actor call log propagation work recursively? (Actor calls an Actor who calls another Actor...)

src/apify_client/clients/resource_clients/log.py

uv.lock

Pijukatel · 2025-05-16T12:16:29Z

Does the Actor call log propagation work recursively? (Actor calls an Actor who calls another Actor...)

That is a really good point. It does if the actors were build with version that includes this change. On the other hand the line breaking does not work as expected anymore as the time marker is present for each log (so with two level of log redirection it will be there 3 times and thus there will be extra split)

See example log from this recursive actor:

Removing timestamp in redirected logs is probably also not OK as those timestamps are important especially for logs that happened before the streaming started.
That is something to figure out before merging this.

Update test data to properly match the reality with line endings

Pijukatel · 2025-05-16T13:12:20Z

Does the Actor call log propagation work recursively? (Actor calls an Actor who calls another Actor...)

I updated the split pattern and test data to properly respect the real data and now it works. See example of 4 level recursive actor and it's logs:

Naming.

vdusek · 2025-05-19T15:58:31Z

tests/unit/test_logging.py

+    mock_api_sync: None,  # noqa: ARG001, fixture
+    propagate_stream_logs: None,  # noqa: ARG001, fixture


Could we do it better here? Maybe pytest-mark-usefixtures.

@janbuchar I could not find the comment, but I remember we were already discussing it some time ago in another review and you were in favor of # noqa.

I have no problem using this fixture and I am also OK with the # noqa, so if you agree with @vdusek on the preferred way in our code base, I will just use it.

I'm mostly OK with both, but usefixtures does have the drawback of not being applicable to other fixtures...

alright, so it's up to you Pepa 🙂

no change then :-)

vdusek

LGTM

### Description - Add convenience methods and arguments for redirecting the streamed actor log to another log. - This changes the default behavior in non non-breaking way. By default, the logs will be redirected now. - Update log endpoints calls to accept `raw` parameter. - Parity with Python implementation apify/apify-client-python#403 ### Example usage: #### Actor.call - default This will redirect logs by default. Example default redirected line: > 2025-10-22T13:06:39.476Z redirect-log-tester runId:SoK1VRJxG61tVrgNz -> 2025-10-22T13:06:06.686Z ACTOR: Pulling container image of build Gh9yAJtjw0rHpc1NU from registry. ```javascript ... await client.actor('redirect-actor-id').call(); ``` #### Actor.call - custom Log This will redirect logs using custom log and logger. Example default redirected line: > 2025-10-22T13:06:39.476Z customPrefix 2025-10-22T13:06:06.686Z ACTOR: Pulling container image of build Gh9yAJtjw0rHpc1NU from registry. ```javascript ... await client.actor('redirect-actor-id').call(someInputs, { log: new Log({ level: LEVELS.DEBUG, prefix: 'customPrefix', logger: new LoggerActorRedirect() }), ``` #### Actor.call - no log redirection This will disable all log redirection (same as current behavior) ```javascript ... await client.actor('redirect-actor-id').call(someInputs, {log: null}), ``` #### Actor.run - attaching to already running actor and redirecting new logs A typical use case is redirecting logs from an Actor that runs in standby. We do not want all the logs; we only want the new logs generated from the moment of connection. ```javascript ... const streamedLog = await client.run('someActorRunId').getStreamedLog({ fromStart: false }); streamedLog.start(); // Do some stuff while also redirecting logs from another Actor await streamedLog.stop(); ``` #### Example actor with recursive redirection: https://console.apify.com/actors/IcfIKTIvbmujMmovj/source ### Issues - Partially implements: #632

Pijukatel added 4 commits May 12, 2025 15:44

TODO: Figure out hwo to mock response with steram

a71ae41

WIP

69ff84c

TODO: Message inspection to split bulk messages TODO: Log level guessing outside of filter

Polish spliting of messages and setting the log level

862cacc

Update test to be able to use caplog

Draft with async implementation and example tests

753427a

Pijukatel added 3 commits May 14, 2025 11:51

Add raw=True

cc0d944

Add chunck processing

cbcabd3

Merge remote-tracking branch 'origin/master' into redirected-actor-logs

81577e8

github-actions bot assigned Pijukatel May 14, 2025

github-actions bot added this to the 114th sprint - Tooling team milestone May 14, 2025

github-actions bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels May 14, 2025

Pijukatel added 2 commits May 14, 2025 17:05

Add sync version of the logging.

b9bc44d

Refactor tests and code to reuse code.

Finalize, update comments

9720327

Pijukatel added 2 commits May 15, 2025 09:21

Add from_start argument for streaming from stand-by actors

85ead2f

Update default logger to not duplicate handlers if already exists

Skip first logs based on datetime of the marker

4ad39fa

Pijukatel requested a review from Copilot May 15, 2025 08:54

Copilot AI reviewed May 15, 2025

View reviewed changes

src/apify_client/clients/resource_clients/actor.py Outdated Show resolved Hide resolved

src/apify_client/clients/resource_clients/log.py Show resolved Hide resolved

Self review.

74595f9

Have explicit start and stop instead of __call__

Pijukatel force-pushed the redirected-actor-logs branch from 32def80 to 74595f9 Compare May 15, 2025 09:12

Pijukatel changed the title ~~Add redirected actor logs~~ feat: Add redirected actor logs May 15, 2025

Pijukatel requested review from MQ37, janbuchar and vdusek May 15, 2025 09:23

Pijukatel marked this pull request as ready for review May 15, 2025 09:27

barjin reviewed May 15, 2025

View reviewed changes

src/apify_client/clients/resource_clients/log.py Outdated Show resolved Hide resolved

janbuchar reviewed May 15, 2025

View reviewed changes

janbuchar self-requested a review May 15, 2025 11:12

MQ37 approved these changes May 15, 2025

View reviewed changes

Handle bytestream edgecase of chunk containing only half of the multi…

cba571f

…byte character

Pijukatel force-pushed the redirected-actor-logs branch from df4afce to cba571f Compare May 15, 2025 12:03

Pijukatel added 2 commits May 15, 2025 14:20

Review comments

02a1eb2

Remove unnecessary actor_name argument

2674cf2

vdusek requested changes May 16, 2025

View reviewed changes

src/apify_client/clients/resource_clients/log.py Outdated Show resolved Hide resolved

uv.lock Show resolved Hide resolved

Update split pattern to deal with multiple times redirected log

2a6f2ec

Update test data to properly match the reality with line endings

Pijukatel added 2 commits May 16, 2025 15:19

Review comment

1263450

Regenerate uv.lock with new version of uv

b1338f1

Pijukatel requested a review from vdusek May 16, 2025 13:33

Test data time alignment.

669a749

Naming.

Pijukatel force-pushed the redirected-actor-logs branch from c29b896 to 669a749 Compare May 16, 2025 13:46

vdusek reviewed May 19, 2025

View reviewed changes

vdusek approved these changes May 20, 2025

View reviewed changes

Pijukatel merged commit fd02cd8 into master May 20, 2025
28 checks passed

Pijukatel deleted the redirected-actor-logs branch May 20, 2025 14:24

Pijukatel mentioned this pull request Oct 22, 2025

feat: Add redirected actor logs apify/apify-client-js#769

Merged

		mock_api_sync: None, # noqa: ARG001, fixture
		propagate_stream_logs: None, # noqa: ARG001, fixture

feat: Add redirected actor logs #403

feat: Add redirected actor logs #403

Uh oh!

Conversation

Pijukatel commented May 13, 2025 • edited by vdusek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Example usage:

Starting new actor run through call

Attaching to already running actor, redirecting all logs from actor start

Attaching to already running actor, redirecting only new logs

Example actor with all the logging options shown:

Issues

Uh oh!

Pijukatel commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pijukatel commented May 14, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Pijukatel commented May 15, 2025

Uh oh!

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

janbuchar May 15, 2025

Choose a reason for hiding this comment

Uh oh!

Pijukatel May 15, 2025

Choose a reason for hiding this comment

Uh oh!

janbuchar May 15, 2025

Choose a reason for hiding this comment

Uh oh!

Pijukatel May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MQ37 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Pijukatel commented May 16, 2025

Uh oh!

Pijukatel commented May 16, 2025

Uh oh!

vdusek May 19, 2025

Choose a reason for hiding this comment

Uh oh!

Pijukatel May 20, 2025

Choose a reason for hiding this comment

Uh oh!

janbuchar May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vdusek May 20, 2025

Choose a reason for hiding this comment

Uh oh!

Pijukatel May 20, 2025

Choose a reason for hiding this comment

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pijukatel commented May 13, 2025 •

edited by vdusek

Loading

Pijukatel commented May 13, 2025 •

edited

Loading

Pijukatel May 15, 2025 •

edited

Loading

MQ37 left a comment •

edited

Loading

janbuchar May 20, 2025 •

edited

Loading